Colorization by Matrix Completion

Given a monochrome image and some manually labeled pixels, the colorization problem is a computer-assisted process of adding color to the monochrome image. This paper proposes a novel approach to the colorization problem by formulating it as a matrix completion problem. In particular, taking a monochrome image and parts of the color pixels (labels) as inputs, we develop a robust colorization model and resort to an augmented Lagrange multiplier algorithm for solving the model. Our approach is based on the fact that a matrix can be represented as a low-rank matrix plus a sparse matrix. Our approach is effective because it is able to handle the potential noises in the monochrome image and outliers in the labels. To improve the performance of our method, we further incorporate a so-called local-color-consistency idea into our method. Empirical results on real data sets are encouraging. Introduction For technical reasons, old photos and films are all monochrome, and it is of great interest to colorize those monochrome images and films. Computer assisted colorization has become an important application of artificial intelligence and has been widely applied to free technicians from manual colorization. Many methods have been proposed for the colorization problem in the literature (Horiuchi 2002; Levin, Lischinski, and Weiss 2004; Yatziv and Sapiro 2006; Luan et al. 2007). One seminal work is the optimization method of Levin, Lischinski, and Weiss (2004). The key idea is based on the assumption that neighboring pixels have similar colors if their intensities are similar. As a result, the colors of unlabeled pixels are estimated by minimizing the difference from the weighted average of the colors at the neighboring pixels. The monochrome pixels are the observations, some of which are labeled with colors and the rest are unlabeled. The task is to learn a function which predicts colors (labels) for the unlabeled pixels. This optimization method uses both labeled and unlabeled pixels for training, thus enjoys semisupervised learning mechanism (Cheng and Vishwanathan 2007). However, the local-color-consistency assumption makes the method of Levin, Lischinski, and Weiss (2004) have Copyright c © 2012, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. (a) Monochrome (b) Labels (c) Recovered Figure 1: Colorization using our method (Low-rank+Localcolor-consistency) with 1% pixels labeled with colors. two major limitations. First, colors are sometimes not local consistent, such as in some complex textures. Second, this local-color-consistency assumption requires each similarcolor patch has at least one labeled pixel. Unfortunately, since similar-color patches are sometimes very small, there are numerous such patches, which makes it hard to guarantee each patch to include one labeled pixel. In this paper we propose a new semi-supervised learning method for tackling the colorization problem. Our work is motivated by the recent advances of matrix recovery and its extensions. Matrix recovery is a class of problems of restoring a matrix corrupted by noises and outliers or a matrix with missing entries. Rank minimization plays a central role in matrix recovery techniques (Candès and Recht 2009; Cai, Candès, and Shen 2010; Mazumder, Hastie, and Tibshirani 2010). In practical applications, as a convex surrogate of the matrix rank, the nuclear norm is typically employed to deal with the NP-hard problem of rank minimization. Recently, Candès et al. (2011) proved that an abitrary matrix can be represented as a low-rank matrix plus a sparse matrix. Accordingly, they proposed the robust principal component analysis (RPCA) model. Owing to the strong theories and tractable computations, matrix recovery has received wide applications in collaborative filtering (Candès and Recht 2009; Cai, Candès, and Shen 2010), background modeling (Candès et al. 2011), subspace clustering (Liu, Lin, and Yu 2010), image alignment (Peng et al. 2010), camera calibration (Zhang, Matsushita, Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence